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\^ ' Abstract 

This paper considers the achievable rates and decoding complexity of low-density parity-check (LDPC) 
codes over statistically independent parallel channels. The paper starts with the derivation of bounds 
^ on the conditional entropy of the transmitted codeword given the received sequence at the output of the 

O ■ parallel channels; the component channels are considered to be memoryless, binary-input, and output- 

symmetric (MBIOS). These results serve for the derivation of an upper bound on the achievable rates of 
ensembles of LDPC codes under optimal maximum-likelihood (ML) decoding when their transmission takes 
place over parallel MBIOS channels. The paper relies on the latter bound for obtaining upper bounds 
, on the achievable rates of ensembles of randomly and intentionally punctured LDPC codes over MBIOS 

I channels. For ensembles of punctured LDPC codes, the calculation of bounds on their thresholds under 

' ML decoding and their exact thresholds under iterative decoding (based on density evolution analysis) is 

■ of interest in the sense that it enables to assess the degradation in the asymptotic performance which is 

attributed to the sub-optimality of iterative decoding (as compared to ML decoding), and also to assess 
the inherent loss in the asymptotic performance which is attributed to the structure of these ensembles, 
even if ML decoding could be applied to decode LDPC codes. The paper also provides a lower bound 
on the decoding complexity (per iteration) of ensembles of LDPC codes under message-passing iterative 
decoding over parallel MBIOS channels; the bound is given in terms of the gap between the rate of these 
I codes for which reliable communication is achievable and the channel capacity. Similarly to the case of a 

single MBIOS channel, this lower bound grows like the log of the inverse of the gap to capacity. The latter 
bound is used for the derivation of lower bounds on the decoding complexity of punctured LDPC codes over 
MBIOS channels; looser versions of these bounds suggest a simplified re-derivation of previously reported 
bounds on the decoding complexity of randomly punctured LDPC codes. The paper presents a diagram 
which shows interconnections between the theorems introduced in this paper and some other previously 
reported results. The setting which serves for the derivation of the bounds on the achievable rates and 
decoding complexity is general, and the bounds can be applied to punctured LDPC codes, non-uniformly 
error protected LDPC codes, and LDPC-coded modulation where all these scenarios can be treated as 
different forms of communication over parallel channels. 

Index Terms: Block codes, complexity, low-density parity-check codes, iterative decoding, 
maximum-likelihood decoding, parallel channels, punctured codes, thresholds. 
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1 Introduction 



This paper focuses primarily on the information-theoretic Umitations of low-density parity-check 
(LDPC) codes whose transmission takes place over a set of parallel channels. By parallel channels 
we mean a communication system or model where the distortion introduced by each channel is 
independent of the signal and the distortion in all of the other channels. The first main result 
presented in this paper is the derivation of an upper bound on the achievable rates of ensembles of 
LDPC codes under optimal maximum-likelihood (ML) decoding when we assume that the codes are 
communicated over parallel memoryless binary-input output-symmetric (MBIOS) channels. This 
result forms a non-trivial generalization of the "un-quantized" bound derived by the authors for 
a single MBIOS channel jl7j . where the latter bound improves the tightness of the bound in ^ 
which was based on a two- level quantization approach. The second main result introduced in this 
paper is a derivation of a lower bound on the decoding complexity of ensembles of LDPC codes 
for parallel MBIOS channels; this bound assumes an iterative message-passing decoder, and the 
decoding complexity is normalized per iteration. In this respect, it continues the study in which 
was focused on upper bounds on the parity-check density of ensembles of LDPC codes for parallel 
MBIOS channels. The two main issues addressed in this paper for parallel channels are applied to 
punctured LDPC codes. 

Puncturing serves to increase the rate of the original code by reducing the length of the code- 
word. The performance of punctured LDPC codes under ML decoding was studied in [2 via 
analyzing the asymptotic growth rate of their average weight distributions and using upper bounds 
on the decoding error probability under ML decoding. Based on this analysis, it was proved that 
for any MBIOS channel, capacity-achieving codes of any rate can be constructed by puncturing 
the code bits of ensembles of LDPC codes with small enough design rate. In [J, the performance 
of punctured LDPC codes was studied under message-passing iterative decoding for the AWGN 
channel. Ha and McLaughlin studied in two methods for puncturing LDPC codes where the 
first method assumes random puncturing of the code bits at a fixed rate, and the second method as- 
sumes possibly different puncturing rates for each subset of code bits which corresponds to variable 
nodes of a fixed degree. In the second approach which is called 'intentional puncturing', the degree 
distributions of the puncturing patterns were optimized in llf 151 were it was aimed to minimize the 
threshold under iterative decoding for a given design rate via the Gaussian approximation; exact 
values of these optimized puncturing patterns were also calculated by the density evolution analysis 
and show good agreement with the Gaussian approximation. The results in |^ [3] exemplify the 
usefulness of punctured LDPC codes for a relatively wide range of rates, and therefore, they are 
suitable for rate-compatible coding. Some fundamental properties of punctured LDPC codes were 
studied in ^2j, and it was specifically claimed that for an arbitrary MBIOS channel and rates 
Ri and R2 satisfying < i?i < i?2 < there exists an ensemble of LDPC codes which can be 
punctured from rate Ri to rate R2 resulting in good codes for all rates Ri < R < R2. This interest- 
ing property implies that rates arbitrarily close to one are achievable via puncturing of a suitable 
ensemble of LDPC codes whose rate (before puncturing) is sufficiently low. These results show 
the high potential of puncturing in designing codes which operate closely to the Shannon capacity 
limit and used for rate-compatible coding for various memoryless binary-input output-symmetric 
(MBIOS) channels. 

The transmission of punctured codes over an arbitrary channel can be treated as a particular 
case of communication of the original code over a set of parallel channels (where the puncturing rate 
of each set of code bits cascaded with the communication channel of the punctured code serve to 
define the set of parallel channels for the transmission of the original code before puncturing) . We 
therefore apply the bounds on the achievable rates and decoding complexity of LDPC codes over 
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statistically independent parallel channels to the case of transmission of ensembles of punctured 
LDPC codes over a single MBIOS channel. We state puncturing theorems related to achievable 
rates and decoding complexity of punctured LDPC codes. For ensembles of punctured LDPC 
codes, the calculation of bounds on their thresholds under ML decoding and their exact thresholds 
under iterative decoding (based on density evolution analysis) is of interest in the sense that it 
enables to assess the degradation in the asymptotic performance which is attributed to the sub- 
optimality of iterative decoding (as compared to ML decoding), and also to assess the inherent loss 
in the asymptotic performance which is attributed to the structure of these ensembles, even if ML 
decoding could be applied to decode LDPC codes. 

The paper is organized as follows: Section [21 presents preliminary material. Section |31 derives 
bounds on the conditional entropy of the transmitted codeword given the received sequence at 
the output of the parallel channels where the component channels are considered to be MBIOS. 
Section |1] relies on the previous bounds and derives an upper bound on the achievable rates of 
LDPC codes under ML decoding for parallel channels. Section El relies on the previous section for 
the derivation of upper bounds on the achievable rates of ensembles of randomly and intentionally 
punctured LDPC codes whose transmission takes place over MBIOS channels, and numerical results 
are exemplified for various ensembles. Section j^l provides a lower bound on the decoding complexity 
of ensembles of LDPC codes under message-passing iterative decoding for parallel MBIOS channels. 
The latter result is used in this section for the derivation of lower bounds on the decoding complexity 
of randomly and intentionally punctured LDPC codes for MBIOS channels; looser versions of these 
bounds suggest a simplified re-derivation of previously reported bounds on the decoding complexity 
of randomly punctured LDPC codes (as shown in an appendix). Finally, Section [3 summarizes our 
discussion, and presents a diagram which shows interconnections between the theorems introduced 
in this paper and some other previously reported results from [Tl ITUl ITTl IT^ ITCl [T7j . 



2 Preliminaries 



Assume that the communication takes place over J statistically independent parallel channels where 
each of the individual channels is an MBIOS channel whose probability density function is given by 
p{'\' i j) (j = 1) 2, ■ ■ ■ , J)- If we let T{j) be the set of indices of the symbols in an n-length codeword 
which are transmitted over the j^^ channel, then 

J 

Pn {y\x) = n n Piyi\^i'^3) ■ (1) 
3=1 

In this paper, we consider the transmission of binary linear block codes over a set of J parallel 
independent MBIOS channels. The results apply both to individual codes and ensembles, and are 
applied to ensembles of LDPC codes. In the following, we briefiy present standard notation used 
for the characterization of ensembles of LDPC codes, and also present some reported results on 
LDPC codes which are related to this paper. 

In general, an LDPC code is a linear block code which is represented by a sparse parity-check 
matrix H. This matrix can be expressed in an equivalent form by a bipartite graph Q whose 
variable nodes (appearing on the left of Q) represent the the code bits, and whose parity-check 
nodes (appearing on the right of G) represent the linear constraints defined by H. In such a 
bipartite graph, an edge connects a variable node with a parity-check node if and only if the 
variable is involved in the corresponding parity-check equation; the degree of a node is defined as 
the number of edges which are attached to it. 
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Following standard notation, let Aj and pi denote the fraction of edges attached to variable and 
parity-check nodes of degree i, respectively. In a similar manner, let Aj and Fj denote the fraction 
of variable and parity-check nodes of degree i, respectively. The LDPC ensemble is characterized 
by a triplet (n, A, p) where n designates the block length of the codes, and the polynomials 



a(x)^x;a, 



iX 



i-1 



p{x) 



i=l 



oo 
i=l 



(2) 



represent, respectively, the left and right degree distributions (d.d.) from the edge perspective. 
Equivalently, this ensemble can be also characterized by the triplet (n. A, F) where the polynomials 



A(x)^X]A,x\ F(x)^X]F, 



(3) 



1=1 



i=l 



represent, respectively, the left and right d.d. from the node perspective. We denote by LDPC(n, A, p) 
(or LDPC(n, A, F)) the ensemble of codes whose bipartite graphs are constructed according to the 
corresponding pairs of degree distributions. One can switch between degree distributions w.r.t. to 
the nodes and edges of a bipartite graph, using the following equations [Tl| : 



A(x) 



\{u)du 



\{u)du 



\{x) 



A'(l) 



p{u)du 

nx) - H 

/ p{u)du 
Jo 

F'(x) 



p{x) 



F(l) • 



(4) 
(5) 



An important characteristic of an ensemble of LPDC codes is its design rate. For an LDPC ensemble 
whose codes are represented by parity-check matrices of dimension cx n, the design rate is defined 



to be i?d = 1 



This serves as a lower bound on the actual rate of any code from this ensemble. 



and is equal to the actual rate if the parity-check matrix of a code is full rank (i.e., the linear 
constraints which define this code are linearly independent). For an ensemble of LDPC codes, the 
design rate can be calculated in terms of the degree distributions (either w.r.t. the edges or nodes 
of a graph) by two equivalent expressions: 



p{x)dx 



\{x)dx 



T'il) 



(6) 



A sufficient condition for the asymptotic convergence of the rate of codes from an LDPC en- 
semble to its design rate was recently stated in (HI lemma 7]. 

Lemma 2.1 (|BJ? Lemma 7). Let C be a code which is chosen uniformly at random from the 
ensemble LDPC(n, A, F), let R be the rate of C, and R^ be the design rate of this ensemble. Consider 
the function 



-A'(l)log2 

oo 

+ log2 



1 + uv 



1=1 



l + u){l 
2(1 + n)' 



A'(l) , 



1 + 



l + v 
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where ^ 

Assume that the function ^ achieves its global maximum in the range u € [0, oo) at u = 1. Then, 
there exists a constant B > Q such that, for any ^ > and n > no(^, A, F), 

PT{\R-Rd\ >e} <e-^"« . 

Moreover, there exists C > such that, for n > no(^. A, F) 

^ r , „ ^ , -, C log n 
E{\R-Rd\} < . 

In Section EH we rely on this lemma in order to verify that the design rates of various ensembles 
of LDPC codes are equal in probability 1 to the asymptotic rates of codes from these ensembles. 



3 Bounds on the Conditional Entropy for Parallel Channels 

This section serves as a preparatory step towards the derivation of upper bounds on the achievable 
rates of ML decoded binary linear block codes whose transmission takes place over statistically 
independent parallel MBIOS channels. To this end, we present in this section upper and lower 
bounds on the conditional entropy of the transmitted codeword given the received sequence at the 
output of these channels. 



3.1 Lower Bound on the Conditional Entropy 

We begin by deriving an information-theoretic lower bound on the conditional entropy of the 
transmitted codeword given the received sequence, when the transmission takes place over a set of 
J independent parallel MBIOS channels. 

Proposition 3.1. Let C be a binary linear block code of length n, and assume that its transmission 
takes place over a set of J statistically independent parallel MBIOS channels. Let Cj denote the 
capacity of the j'*^ channel (in bits per channel use), and a{-;j) designate the conditional pdf oi the 
log- likelihood ratio (LLR) at the output of the j^^ channel given its input is 1 (where j G {1, . . . , J} 
and the channel input can be either ±1). Let x = (xi, . . . x„) and y = (yi, . . . , y„) designate the 
transmitted codeword and received sequence, respectively, T{j) be the set of indices of the code bits 
transmitted over the j^^ channel, nl-'l = \T{j)\ be the size of this set, and pj = be the fraction 
of bits transmitted over the j*^ channel. For an arbitrary c x n parity-check matrix H of the code 
C, let (3j^rn designate the number of indices in referring to bits which are involved in the m}^ 
parity-check equation of H (where m G {1, . . . , c}), and let i?d = 1 — :^ be the design rate of C. 
Then, the conditional entropy of the transmitted codeword given the received sequence satisfies 

™ >- l-Ef.C.-fl-^^) - 2n(l-'i^)ln2 S{^Y'nfa..)^'4) in 

j=l \ p=l m=l j=l J y 

where 

gj,p = J^°° a{l;j){l + e-')tanh^p(^^^dl, jG{l,...,J}, p G N. (8) 
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Proof. The proof relies on concepts which are presented in |^ El j and generahzes them to the case 
of parallel channels. Let us assume that x G C is the transmitted codeword, and that the '0' and '1' 
symbols are mapped to +1 and —1, respectively. The input alphabet of each component channel 
is {+1, —1}, and if a symbol is transmitted over the j**^ MBIOS channel and y is its corresponding 
output, then the LLR gets the form 

where y denotes the output alphabet of each component channel,^ andp(-|-; j) is the conditional pdf 
of the j*^ channel. For each one of these J component channels, we move from the original mapping 
of X — > y (where according to (^, each symbol is transmitted over only one of these J channels) 
to an equivalent representation of the channel X ^ Y, so that H{X\Y) = H{X\Y). The basic 
idea for showing the equivalence between the original channel and the one which will be introduced 
shortly is based on the principle that the LLR forms a sufficient statistics of an MBIOS channel, 
and the LLR is an anti-symmetric function of the channel output (i.e., LLR(— y; j) = — LLR(y; j) 
for j = 1, . . . , J). This yields that the absolute value of the LLR doesn't change when the channel 
output is flipped, but the sign of the LLR alternates. 

In the following, we characterize an equivalent channel to the considered set of J parallel 
channels. For each index i € let us choose independently a value li according to the conditional 
pdf a{-;j), and for i E {1, . . . ,n}, let 



t^i — \h\, 



+1 if li>0 

-1 if /i < . 

±1 w.p. i if li = 

The output of the equivalent channel is defined as y = (yi, . . . , y„) where yi = ((/>«, Wj) and (pi = 9iXi. 
This defines the mapping 

X ^Y = {^,Vt) 

where $ is a binary random variable which is affected by X, and 17 is a non-negative random 
variable which represents the absolute value of the LLR. For each index i € the pdf oi ilj is 

independent of i, and it gets the form 

\ a{u-j)+a{-uj;j) = {l + e-^)a{u:]j) if > 
fnM = fn{u-j) = ^ a(0;j) if = ' 

We note that the transition in case w > follows from the symmetry property of a{-;j), and that 
the random variable fij is statistically independent of Xi (where i G {1, . . . ,n}). 

Denoting by R the rate of the code C, since the codewords are transmitted with equal probability 

H{X) = nR. (10) 
Also, since the J parallel channels are memoryless, then 

n 

HiY\X) = ^H{Yi\X,). (11) 

i=l 



^In case the output alphabets of the component channels are not equal, then y can be defined as their union. 
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The mapping 1^ ^ 1^ is memoryless, hence i?(Y|Y) = X]r=i -^(^1^)) ^^'^ 

H(Y) = H{Y) - H{Y\Y) + H{Y\Y) 

n 

= H{Y)-^H{Yi\Yi) + H{Y\Y) (12) 



i=l 



H{Y\Y) < ^i?(yi|y,) 

i=l 
n 

= Y.[HiY,)-H{%) + Hi%\Y)]. (13) 



i=l 



Applying the above towards the derivation of a lower bound on the conditional entropy H(K\Y) 
we get 



ii'(X|Y) = H{X) + H{Y\X) - H{Y 

n. 

(a) 

= nn -\- 

i=l i=l 

(6) 



nR + Y,H{Yi\Xi) - H{Y) - H{Y\Y) + Y,H{Yi\Yi) 

i=l i=l 

> nR + Y,H{Yi\Xi)-H{Y)-Y,[H{Y^)-H{Yi) + H{%\Yi)\ +Y,H{Yi\Yi] 

i=l i=l i=l 

n n 

nR - H{Y) + J2h{%) - - HinXi)] 

1=1 1=1 

n n 

nR - HiY) + H{Yi) - ^ /(X^; Yi) 

i=l i=l 
n J 



> nR - H{Y) + H{Yi) - ^ n^^^ Cj (14) 
i=i j=i 

where (a) relies on (fTUIl - lfT^ . (b) rehes on ((O)) . and (c) follows since I{Xi; Yi) < Cj for all i G 
and = n^-'^ for j S {1, . . . , J}. In order to obtain a lower bound on /f(X|Y) from (|14jl . we will 

calculate the exact entropy of the random variables {1^}, and find an upper bound on the entropy 
of the random vector Y. This will finally provide the lower bound on the conditional entropy given 
in ((TJ. Considering an index i £ for some j € {1, 2, ... J}, we get 

HiYi) = Hi^uni) 

= H{ni) + H{<!>i\Qi) 

= H{ni) + E^[H{'^i\ni = u;)] 

= H{ni) + i (15) 

where the last transition is due to the fact that given the absolute value of the LLR, since the 
parallel channels are MBIOS and the coded bits are equally likely to be or 1, the sign of the 
LLR is equally likely to be positive or negative. The entropy H{i^i) isjiot expressed explicitly as 
it cancels out later. In the following, we derive an upper bound on H(Y): 

H{Y) = //(($!,...,$„), {ni,...,Qn)) 

= H{ni, ...,nn) + H{{<^i, ...,$„) I (i7i, ... , nn)) 

n 

= J^i^(l^,)+^((^l,---,^n) I (f^l,...,^^n)) (16) 
1=1 
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where the last equahty follows since the random variables are statistically independent. 



Let us introduce the assignment / : {+1, —1} {0, 1} which maps back the +1 and —1 to 
and 1, respectively, and let <I>j = f{^i)- Since = @iXi, then 



1, 



,n 



where the last addition is modulo-2. Define the syndrome vector 

S = 

where H is a c x n parity-check matrix of the binary linear block code C, and let L be the index 
of the vector ($i, . . . , in the coset which corresponds to S. Since each coset has exactly 2^^ 
elements which are equally likely then H(L) = nR, and we get 

< H{L) + H{s\ini,...,nn)) 
iR + H{s I {ni,...,nn)) 



nl 



< nR+Y,H{Srn\[^l,....^n)). 



(17) 



m=l 



Since (xi, . . . ,x„) denotes a selected transmitted codeword in C (i.e., before converting its symbols 
to ±1 and their transmission over the parallel channels), then 

S = ($i,...,$„)i/^ 

= (ei,...,e„)i7^ + (Xi,...,x„)F^ 
= (ei,...,e„)/?^. 

Let us look at the m-th parity-check equation which involves km variables, and assume that the set 
of indices of these variables is {ii, . . . , ifc^}. Then, the component Sm of the syndrome is equal to 1 
if and only if there is an odd number of ones in the random vector {Qi^, ■ ■ ■ , Gj^.^). To calculate 
the probability that Sm is equal to 1, we rely on the following lemma: 

Lemma 3.1 ( [17] , Lemma 4.1). If the m-th linear constraint defined by the parity-check matrix 
H involves km variables, and if {ii, . . . ,ik^} denote the indices of these variables, then 



Pr(S',„ = 1 I (fiij , . . . , f^i^^ ) = («!,..., Ofe^)) 



tanh 



10 = 1 



(18) 



From this lemma, we obtain 



H [Smli^h,- ■ • ,f^ife„) = ("1, - • • >"fc™)) = ^2 



1 — tanh 



10 = 1 



a,, 



where /i2(') denotes the binary entropy function to base 2. By taking the statistical expectation 
over the km random variables ^li^ , ■ ■ ■ , ^i,. , we get 



oo />oo 



JO 



1 _ JJ tonh 



10 = 1 



JJ h,^ {oiw) daida2 ■ ■ ■ duk^ . 



w=l 
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Let Pj^m denote the number of indices w € {ii, . . . , ikm} referring to variables which are transmitted 
over the j*^ channel. Prom the Taylor series expansion of /i2(') around ^ (see 1171 Appendix C.2]) 



h2ix) = 1 



(1 - 2x)^t 



^ oo 

2 In 2 ^ p(2p - 1) 
p=i 



< X < 1 



(19) 



it follows [m Appendix C.3] that 

H{Smm,,,...,ni^j) 

= 1 - 



1 



b I {s^n (I 



(20) 



where the first transition is based on ((T^ and goes along the same lines as ^1 Appendix C.3]), 
and the second transition is due to the fact that for all % € 2r(j), the pdf oi the random variable Jlj 
is independent of i, see ©. Summing over all the parity-check equations of H gives 

c 



m=l 



^ »f2p- 1) 



m=l 



/n(«;j) tanh^P 



/3j,n 



2 In 2 ^p(2p 

By combining Q, (fTKI) . (fTTj) and (|21|). we get the following upper bound on H{Y) 

n 

H{Y) < ^H{ni) +nR 



(21) 



i=l 



+ C 



1 ^ 1 



2cln2 



^p(2p 



TyE n 

^ m=l I i=l 



a(a;j)(l + e-")tanh2p(|jda 



(a) 



^ i?(J7i) + ni? + n(l - Rd) 



i=l 



y 

2n(l - i?d)ln2 ^ 

p=i 



(22) 



where (a) relies on the definition of gj^p in Q and since i?d — 1 ~ « denotes the design rate of 
C. Finally, the substitution of H15|) and H22j) in the RHS of H14j) provides the lower bound on the 
conditional entropy ff(X|Y) given in ((2)). This completes the proof of the proposition. □ 



3.2 Upper Bound on the Conditional Entropy 

In this section, we provide an upper bound on the conditional entropy of a transmitted codeword 
given the received sequence. The bound holds for an arbitrary binary linear block code whose 
transmission takes place over a set of parallel channels, and is expressed in terms of the code rate 
and the bit-error probability of the code which is associated with an arbitrary decoder. 
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Lemma 3.2. Let C be a binary linear block code of length n and rate R, and assume that its 
transmission takes place over a set of parallel channels. Let x = (xi, . . . , Xn) and y = (yi, . . . , y^) 
designate the transmitted codeword and the received sequence, respectively. Then 

^^^'^^ <fi/i2(Pb) (23) 
n 

where Pb designates the bit error probability of the code C. 

Proof. Since there is a one to one correspondence between the codewords and the set of information 
bits used to encode them, then /f(X|Y) = i?(U|Y) where the vector u = (ui, . . . ,UnR) denotes 
the sequence of information bits used to encode the codeword x. Let Pj^*^ (C) denote the probability 
of decoding the bit Ui erroneously given the received sequence at the output of the set of parallel 
channels, then the bit error probability is given by 

^ nR 

W = ;^E^b'^(C)- (24) 
1=1 

This therefore gives 

H{X\Y) H(U\Y 



n 



(a) 
< 



ib) 
< 



< Rh 




^ nR \ 
1=1 / 



id) 



Rh2{PbiC)) 



where inequality (a) holds from the chain rule of the entropy and since conditioning reduces entropy, 
inequality (b) follows from Fano's inequality and since the code is binary, inequality (c) is based 
on Jensen's inequality and the concavity of /i2(")) equality (d) follows from (|24|) . □ 



4 An Upper bound on the Achievable Rates of LDPC codes over 
Parallel Channels 

In this section, we derive an upper bound on the design rate of a sequence of ensembles of LDPC 
codes whose transmission takes place over a set of statistically independent parallel MBIOS chan- 
nels, and achieves vanishing bit error probability under ML decoding. This bound is used in the 
next section for the derivation of an upper bound on the design rate of an arbitrary sequence of 
ensembles of punctured LDPC codes. 

Let us assume that a binary LDPC code C of length n is transmitted over a set of J statistically 
independent parallel MBIOS channels. Denote the number of code bits of C which are transmitted 
over the j**^ channel by n^^\ and the fraction of bits transmitted over the j^^ channel by 

Pj^—, iG{l,...,J}. (25) 
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Let G he a. bipartite graph which represents the code C, and E be the set of edges in g. Let 
designate the set of edges connected to variable nodes which correspond to code bits transmitted 
over the j^^ channel, and 

9i = ^, j^{h---,J} (26) 

denote the fraction of edges connected to these variable nodes. Referring to the edges from the 
subset E^^\ let A^^ designate the fraction of these edges which are connected to variable nodes of 
degree i, and define the following J degree distributions from the edge perspective: 

oo 



i=2 



which correspond to each of the J parallel channels. According to this notation, the number of 
edges connected to variable nodes corresponding to code bits transmitted over the j^^ channel is 
given by 

nb1 



m = ^—^, J €{!,..., J}. (27) 

i 

For the simplicity of the notation, let us define a vector of degree distributions for the variable 
nodes from the edge perspective to be \{x) = (AW(x), . . . , A['^1(x)). Following the notation in fH] . 
the ensemble {n,X,p) is defined to be the set of LDPC codes of length n, which according to their 
representation by bipartite graphs and the assignment of their code bits to the parallel channels, 
imply left and right degree distributions of A and p, respectively. 

Lemma 4.1. 

1 J ( ^ 



j^\{x)dx ^\j',m\x)dxy ^^^^ 

where A is the original left degree distribution which serves to construct the vector of left degree 
distributions A, due to the transmission of the LDPC code over parallel channels. 

Proof. Since E^^\ . . . ,E^'^^ forms a sequence of disjoint sets whose union is the set E, we get the 
equality \E\ = Y,i=i \E^^\ From we therefore get 



and by dividing both sides of the equality by n and using (|25j) . the lemma follows immediately. □ 
Lemma 4.2. 

= ,1,,^; , --r-p ^ ^' VjG{l,...,J}. (30) 

^l/^W(x)dx/ 

Proof. The lemma follows directly from (|26j) . (|27|) and Lemma l4. 11 □ 
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In the following, we introduce a sequence of ensembles of LDPC codes, say {(jT-n Ar) Z')}^! 
where the fraction of code bits assigned to each of the J parallel channels is uniform over all the 
codes of each ensemble, and p is fixed for all the ensembles of this sequence (i.e., it is independent 
of r). Since A which corresponds to the overall left degree distribution of the edges is independent 
of r (and so is the right degree distribution p), one can consider here the common design rate of 
the sequence of ensembles {(ra^, A^,, p)}^^]^ which does not depend on r. 

This setting is general enough for applying the following theorem to various applications which 
form particular cases of communication over parallel channels, e.g., punctured LDPC codes 
non-uniformly error protected LDPC codes and LDPC-coded modulation (see e.g., [HI El)- In 
this setting, the fraction of code bits assigned to the j^^ channel, pj^r, depends on j € {1, ... , J} and 
r € N, but not on the particular code chosen from each ensemble. It follows from Lemma 14.21 that 
the same property also holds for Qj^^ which designates the fraction of edges which are connected to 
variable nodes whose code bits are assigned to the j^^ channel. In the following, we assume that 
the limits 

Pj = lim pj r, Qj = lim g,- ^ (31) 
exist and are positive for all j € {1, . . . , J}. 

Theorem 4.1. Let a sequence of LDPC ensembles {{rir, ^^r^ P)}r^i be transmitted over a set of 
J statistically independent parallel MBIOS channels, and assume that the block length (n^) goes 
to infinity as we let r tend to infinity. Let Cj denote the capacity of the j*^ channel, and a{-;j) 
designate the pdf of the LLR at the output of the j^^ channel given its input is 1. If in the limit 
where r tends to infinity, the bit error probability of this sequence under ML decoding vanishes, 
then the common design rate of these ensembles satisfies 

J 

Rd<l -J (32) 

where T, as introduced in Q, denotes the right degree distribution from the node perspective, and 
Qj^p is introduced in (jHJ. 

Proof. Let {Cr}^i be a sequence of LDPC codes chosen uniformly at random from the sequence 
of ensembles {{rir, X^.^ P)}?^i- Denote the rate of the code Cr by Rr, and let Pb,r be its bit error 
probability under ML decoding. Let Qr be a bipartite graph of the code Cr whose left and right 
degree distributions from the edge perspective are A,, and p, respectively. From Proposition 13.11 
and Lemma 13.21 it follows that the following inequality holds for the binary linear block code Cr- 

J 

i=i 

where rir is the block length of the code Cr, Rd is the asymptotic design rate for all the codes 
from the sequence of ensembles {{nr, Xri P)}?^ii ^^'^ Pr,j,m denotes the number of edges which are 
connected to the m*^ parity-check node of the graph Qr and are related to code bits transmitted 
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over the j^^ channel (where j G {1, . . . , J} and m G {1, . . . nr{l — R^)})- By taking the expectation 
on both sides of (|H3|) and letting r tend to infinity, we get 



l-Y,PjC,-il-RA) 



n,(l-Rd) 



• lim I 1 

r— »oo 



2n. 




< 0. (34) 



The last result follows from the LHS of (|33j) . due to the concavity of the binary entropy function 
and Jensen's inequality, and since by our assumption, the bit error probability of the ensembles 
vanishes in the limit where r tends to infinity. 

The derivation of an upper bound on the design rate is proceeded by calculating the expectation 
of the product inside the LHS of 1)34(1 . Let kr^m denote the degree of the m}^ parity-check node of 
the bipartite graph Qr, then the smoothing theorem gives 



E 



E I n (9j,p) 



(35) 



where the outer expectation is carried over the random variable kr^m- We first calculate the inner 
expectation in the RHS of (jSHI)- It follows from (HTJ that the number of edges, \eI^^\ = |Sb](a^)|, 
connected to variable nodes corresponding to code bits transmitted over the j^^ channel, is inde- 
pendent of the code Cr we choose from the ensemble {nr,\,p). The same property also holds for 

the total number of edges in the graph (since l-E^I = S/=i l-^r''l)- Since the code Cr is chosen 
uniformly at random from the ensemble, it follows that if kr^m is a given positive integer, then 



E I n (9j,p) 



E 



bi, . . . ,bj > 



Pr (/?rj,m = 6j , Vj G {1, . . . , J}) {gj^j, 



E 

bi, . . . ,bj >( 



('fV-..-('f|) ' 



(f'') 



n 



(36) 



Lemma 4.3. 



lim 



('tV...^('f') 



Hig.f^ lim ( 



(37) 



Proof. By assumption, in the limit where we let r tend to infinity, the block length rir also tends 
to infinity. Hence, from ((^7)) . we get that for all j G {1, . . . , J}, also tends to infinity in the 
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limit where r tends to infinity. 



lim 

1 — >oo 



(f) 



lim 



\eI^^\\...\eI-^^\\ 



lim 



I^W I !... 1^1.-^11! (|^,|-A:,,„)! 1 .. / k 



\Er\l 



lim 



1 - . . . i\Ey^ \-bj)\\ ™ \bi,b2, ...,bj 



= iim : -, lim 



(b) / \Er 

= iim 



bi 



(c) 



r^oo \ \Er 
J 



r^oo \bi,b2,...,bj 

\Er lim ( ^r,m 

\Er\ / r^oo \ bi,b2,. ■ . ,bj 



lim "["[(g-ir)''^ lim 



r-^'ci. \bi,b2,--- ,bj 



= ll{q,P lim ( 

where equality (a) follows since for all j e {I,..., J}, \eI^^\ ^ oo as we let r tend to infinity, while 
on the other hand, the maximal right degree (and hence, also bi, . . . ,bj and kr^m) stay bounded; 
equality (b) is valid due to the constraint Ylj=i — kr,m, and equality (c) follows from (j^H). □ 



By letting r tend to infinity in both sides of (|35() . and substituting H36|) and (|37() in the RHS of 
p5|) . we get that for all p G N 



hm E 



n (^.. 



(a) 



(b) 



E 



E 



lim E TT (5,,, 



J 



^ ^ Pr,j,m — 



U = l 



lim > 

r— >oo ^— ' 



bl bj > 



kr,m 
61,62, • • • 



E 



(c) 



"-c.max ^ J 

k=i ^ j=i 



(38) 
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where equality (a) follows from (|35() and since the right degree distribution is independent of r (note 
that the outer expectation in equality (a) is performed w.r.t. the degree of the m^^ parity-check 
node); equality (b) follows from H36|) and H37() . and since the number of terms in the sum is bounded 
(this number is upper bounded by {kr^m)'''~^, so it is bounded for all r G N due to the fact that 
the maximal right degree is fixed), and equality (c) follows since the right degree distribution is 
independent of r. Since the limit in (|38j) does not depend on the index m which appears in the 
inner summation at the LHS of ()34() and also limj.^00 'T'r(l — -^d) = 00, then we get from H38|) 



lim — r- 

r-»oo nr{l — Rd) 



nr(l-Rd) 



m=l 



n ^3j,p 



(a) 



(b) 



hm E 



J 



n ^3j,v) 



(39) 



where equality (a) follows from the fact that if {a^} is a convergent sequence then the equality 
lim.f.^00 7 Yl\=i — liiHr^oo Or holds, and also since any sub-sequence of a convergent sequence 
converges to the same limit as of the original sequence; equality (b) follows from H38() . Combining 
(jHH) and JHH) gives 



J 



1 - > V.C, - (1 - i?d) ( 1 - ^ > { TT r( > q,q,,r, 1 H < 0. 

Finally, solving the last inequality for gives the upper bound on the design rate in 1)32(1 . 



□ 



Example 4.1. For the particular case where the J MBIOS parallel channels are binary erasure 
channels where the erasure probability of the j'*^ channel is e,- , we get from (jH| 



9j,p 



1 



Vj G {1,..., J}, pGN. 



(40) 



Since gj>p is independent of p for a BEC, and based on the equality Yl^=i 2p(2p-i) ~ ^"0.2, we 
obtain from Theorem l4.1l that the common design rate of the sequence of LDPC ensembles is upper 
bounded by J 



i=i 



Rd < 1 



This particular result coincides with jlH Theorem 2]. 



1 - r 1 



J 

E«J- 



(41) 



The proof of Theorem 14.11 relies on the assumption that the right degree distribution p is fixed, 
and does not depend on the ordinal number r of the ensemble. For a capacity-achieving sequence 
of LDPC ensembles, both the maximal and the average right degrees tend to infinity (see |15| 
Theorem 1]). Hence, for a capacity-achieving sequence of LDPC codes, p cannot be fixed. 

Remark 4.1. We wish to discuss a possible refinement of the statement in Theorem 14.11 Let us 
assume that the (overall) degree distributions A and p are fixed, but due to the transmission over 
parallel channels, the corresponding vector of degree distributions A,. = (A^^^, . . . , Ai"^') and also pj^r 
and Qj^r depend on the code from the ensemble (n^, A, p). Since the derivation of this theorem relies 
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on the bounds on the conditional entropy from Section [3 (which are vahd code by code), one can 
refine the statement in Theorem 14.11 so that the modified theorem permits the dependency of the 
vector (Ai^', . . . , X^/^) on the specific code chosen from the ensemble. In this case, the equalities in 
(|^T|) are transformed to 

Pj = lim E[p,>(C)] , Qj = lim E [(/,>(C)] 

r^oo 1 — >oo 

where the expectation is carried over the code C from the ensemble {nr,X,p)- In this case, the 
proof of Theorem 14.11 involves an expectation over C on both sides of ()33() (which is valid code by 
code) and then we let r tend to infinity, as in (jHlJ. By invoking Jensen's inequality, Lemma H.'.M is 
changed under the above assumption to the inequality 



lim Ec 



(f) 



>lliqjP lim 



kr,m 

bi,b2,...,bj 



and correspondingly, (jSHJ is changed to 



lim Ec 

r— »oo 

Therefore, the upper bound on the design rate in (|.S2|) holds for the more general setting as above. 



5 Achievable Rates of Punctured LDPC Codes 

In this section we derive upper bounds on the achievable rates of punctured LDPC codes whose 
transmission takes place over an MBIOS channel, and the codes are ML decoded. The analysis in 
this section relies on the bound presented in Section^ 

Let C be a binary linear block code. Assume its code bits are partitioned into J disjoint sets, 
and the bits of the j^^ set are randomly punctured with a puncturing rate ttj (where j € {1, . . . , J}). 
The transmission of this code over an MBIOS channel is equivalent to transmitting the code over 
a set of J parallel MBIOS channels where each of these channels forms a serial concatenation of a 
BEC whose erasure probability is equal to the puncturing rate ttj, followed by the original MBIOS 
channel (see e.g., H UHl CH CI] ) . 



5.1 Some Preparatory Lemmas 

This sub-section presents two lemmas which are later used to prove results for ensembles of randomly 
and intentionally punctured LDPC codes (denoted by RP-LDPC and IP-LDPC codes, respectively). 

In the following lemma, we consider a punctured linear block code and provide an upper bound 
on the conditional entropy of a codeword before puncturing, given the received sequence at the 
output of the channel. This upper bound is expressed in terms of the bit error probability of the 
punctured code. 

Lemma 5.1. Let C be a binary linear block code of length n and rate R', and let C be a code 
which is obtained from C by puncturing some of its code bits. Assume that the transmission of 
the code C takes place over an arbitrary communication channel, and the code is decoded by an 
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arbitrary decoding algorithm. Let x' = {x[, . . . , x^) and y = (yi, . . . , ?/„) designate the transmitted 
codeword of C and the received sequence, respectively. Then, the conditional entropy of the original 
codeword of C given the received sequence satisfies 



n 



(42) 



where Pb designates the bit error probability of the punctured code C. 



Proof. The proof follows directly from Lemma 13.21 and the equivalence between the transmission 
of punctured codes over an MBIOS channel and the particular case of transmitting this code over 
a set of parallel channels (see the introductory paragraph of Section E)) . □ 



Puncturing serves to increase the rate of the original code by reducing the length of the code- 
word. It may however cause several codewords to be mapped onto a single codeword, thereby 
reducing the dimension of the code. Consider a binary linear code, C, of length n and rate R' and 
assume a fraction 7 of its code bits are punctured. In the case that the dimension is not reduced 
by puncturing, the rate of the punctured code is given hy R = In the general case, we cannot 
guarantee that the dimension of the code is not reduced. However, for a sequence of punctured 
codes whose bit error probability vanishes as the block length of the codes goes to infinity, the 
following lemma shows that the rate of the punctured codes converges to the desired rate R. 

Lemma 5.2. Let {C^} be a sequence of binary linear block codes of length and rate R'^., and 
let {Cr} be a sequence of codes which is obtained from {C^} by puncturing a fraction 7 of the code 
bits. Assume the sequence of punctured codes {Cr} achieves vanishing bit error probability in the 
limit where we let r tend to infinity. Then, the asymptotic rate R of the sequence of punctured 
codes is given by 

^ = (43) 
1-7 

where R' = lim^^oo R'r is the asymptotic rate of the original sequence of codes {C^}. 



Proof. Let = (x'l, . . . , x^^.) and y,. = [ur ■ ■ ■ , fjur) designate the original codeword (before punc- 
turing) and the received sequence (after puncturing) , respectively. Since we assume the there exists 
a decoding algorithm such that the punctured codes achieve vanishing bit error probability, we have 
from lemma that 

Let Xr = (xi, . . . , a;„^) designate the codeword after puncturing (where the punctured bits are 
replaced by question marks). Since =^ X,. ^ Y,. forms a Markov chain, then by the informa- 
tion processing inequality, we get //(X^jX^) < //(X^, [Y^). The non- negativity of the conditional 
entropy therefore yields that 

hm ^l^Z^ = 0. (44) 

Denote the dimension of the codes C'^ and Cr by d'^ and d,., respectively. Since C'^ is binary and 
linear, every codeword of Cr originates from exactly 2'^'-"'^'' different codewords of C^. The codewords 
are assumed to be transmitted with equal probability, and therefore i/(X'^|Xr) = d', — dr- Let Rr 
designate the rate of the punctured code Cr- By definition, d'r = P'rUr, and since 71^(1 — 7) forms 
the block length of the punctured code C^, then dr = Rrnr{l — 7). Substituting the last three 
equalities into (|44|) gives 

lim (R'-Rril--/)) =0. 

r— >oo ^ ^ 

This completes the proof of the lemma. □ 
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For a sequence of codes {C^}, it is natural to refer to their code rates R'^. However, for sequences 
of ensembles, where parity-check matrices are randomly picked, such matrices are unlikely to be 
full rank. Hence, a more natural approach is to refer to their design rates. To this end, we define 
the design rate of codes which are obtained by puncturing some code bits of binary linear block 
codes. 

Definition 5.1. Let C be a binary linear block code of length n, H' he, a, cxn parity-check matrix 
of C and -Rrf — 1 — f designate the design rate of the code C . Let C be a code which is obtained 
from C by puncturing a fraction 7 of the code bits. The design rate of C is defined as 

i?d ^ (45) 
1-7 

Prom Lemma 15. 2| it follows that for an arbitrary sequence of punctured codes which achieves 
vanishing bit error probability, their asymptotic design rate is equal in probability 1 to their asymp- 
totic rate if and only if this condition also holds for the original sequence of codes before their 
puncturing. For un-punctured ensembles of LDPC codes, a sufficient condition for the asymptotic 
convergence of the rate to the design rate is introduced in jS| Lemma 7] (see Lemma f2.1l in Section|21). 
In Section [5. 4( we apply this lemma to show that the bounds on the achievable rates of ensembles 
of punctured LDPC codes apply to their actual code rates and not only to their asymptotic design 
rates. 



5.2 Randomly Punctured LDPC Codes 



In this section, we consider the achievable rates of randomly punctured LDPC (RP-LDPC) codes. 
We assume that the transmission of these codes takes place over an MBIOS channel, and refer to 
their achievable rates under optimal ML decoding. The upper bound on the achievable rates of 
ensembles of RP-LDPC codes relies on the analysis in Section 0] where we derived an upper bound 
on the achievable rates of LDPC codes for parallel channels. 

In the following, we assume that the communication takes place over an MBIOS channel with 
capacity C, and we define 



gp = a{l) (1 + e"') tanh^P (^^^ dl , p G N 



(46) 



where a designate the pdf of the LLR of the channel given that its input is zero. 



Theorem 5.1. Let {{rir, X, p)}^i be a sequence of ensembles of LDPC codes whose block length 
(rir) tends to infinity as r — > 00. Assume that a sequence of ensembles of RP-LDPC codes is 
constructed in the following way: for each code from an ensemble of the original sequence, a subset 
of arir code bits is a-priori selected, and these bits are randomly punctured at a fixed rate (i'pct)- 
Assume that the punctured codes are transmitted over an MBIOS channel with capacity C, and 
that in the limit where r approaches infinity, the sequence of ensembles of RP-LDPC codes achieves 
vanishing bit error probability under some decoding algorithm. Then in probability 1 w.r.t. the 
random puncturing patterns, the asymptotic design rate (i?d) of the new sequence satisfies 



Rd < 



1 - aR 



pet 



1 - (1 - aPpct)C 



E 



p=i 



21n2 ^ [pi2p - 1 



r (l-Ppet + e)ft 



(47) 
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where F, as introduced in ©, denotes the right degree distribution (from the node perspective) of 
the original sequence, gp is introduced in (|46j) . and ^ is the following positive number: 



e = 2(1 - a)Ppct / A(x) dx. 



Jo 



(48) 



Proof. By assumption, we select a set of code bits whose size is a fraction a of the code bits, 
and these bits are randomly punctured at rate Ppct- The transmission of the resulting codeword 
over an MBIOS channel is equivalent to the transmission of the original codeword over a set of 
J = 2 parallel channels. The first channel, referring to the set of code bits which are randomly 
punctured, is a serial concatenation of a EEC with erasure probability Ppct and the original MBIOS 
channel; the second channel which refers to the rest of the bits (which are transmitted without 
being randomly punctured) is the original MBIOS channel. For simplicity, let us first assume that 
the degree distribution associated with the selected subset of aur code bits which are randomly 
punctured is independent of the specific code from the ensemble (n^. A, p). Based on the discussion 
above and the notation in Section \^ the transmission of the code bits over these two parallel 
channels induces a sequence of ensembles of LDPC codes, {{ur, Xr, p)}'^i, where = (Ai^',A^) 
depends on the selection of the subset of arir code bits which are randomly punctured. Following 
this equivalence, we get from the notation in Theorem 14 . 1 1 that 



In order to apply Theorem 14. II to our case, we find a global lower bound on the sum Ylj=iQj9j,p 
which does not depend on the a-priori selection of the subset of code bits which are randomly 
punctured. From (jH)) and (|l6|) . it follows that for all p E N: 



Pi = a, p2 = l-a, Ci = C(l - Ppct), C2 = C 



J 




(49) 




and 5'2,p = 9p- Based on Lemmas 14. II and 14.21 we get that for all p E N 




(50) 



where the following constraint is satisfied: 



a 



1 — a 



1 



(51) 




+ 



and 




(52) 
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due to the fact that A[^](x) < x and )S^\x) < x for x G [0, 1] (even without exphcitly knowing At^^ 
and which depend on the a-priori choice of the subset of bits which are randomly punctured). 
Based on we get 



(li9i,p + Q292,p 



(1 - Ppct)gp / A(a;) dx 
Jo 



\ 



a 



AW(x)da 



+ 



1 — a 



Xf\x)d2 



+ 



(1 - a)Ppctgp j 


A(a;) dx 




J 


f APl(x)(ix 





(1 - a)Ppct5p j 


A(x) dx 




J 


f X^\x)dx 





(1 - Ppct)9p + 



> (l - i^pct + 2(1 - a)Ppct ^ A(x) dx^ gp 

= (1 - Ppet + 09p 

where ^ is defined in (|48|) . Since the degree distribution F is a monotonic increasing function, then 



r \J2 i^9,,pj > r ((1 - Ppct + 09p) ■ (53) 

By substituting (|49j) and (|53|) in the RHS of (|32j) . we obtain the following upper bound on the 
asymptotic design rate of the original sequence 

1 - (1 - aPpct)C 



1 °° f 1 1 



Since as r ^ oo, in probability 1 w.r.t. the puncturing patterns, a fraction 7 = aPpd of the code 
bits are punctured, then the asymptotic design rate (Rd) of this sequence satisfies the equality 



Rd 



R'. 



1 - aPr 



(54) 



pet 



from which the theorem follows. 

For the case where the degree distribution associated with the subset of code bits which are 
randomly punctured depends on the code C from the ensemble (n^, A, p), the pair {xl^\ A^) cannot 
be considered to be uniform over all the codes from this ensemble. In this case. Theorem 14. II is not 
directly applicable. In order to circumvent the problem, we rely on the discussion in Remark 14.11 
and on the fact that the lower bound on qiQi^p + q292,p which is given above in terms of ^ from H48() 
is universal for all the codes from this ensemble (i.e., it only depends on A, but does not depend on 
the specific degree distributions Ai^' (C) and A^ (C) which are associated with the code C from the 
ensemble). In light of this reasoning, the proof of the theorem for ensembles of RP-LDPC codes 
also follows in the more general setting where the degree distribution associated with the subset of 
the code bits which are randomly punctured depends on the specific code from the ensemble. □ 



5.3 Intentionally Punctured LDPC Codes 

In 1', Ha and McLaughlin show that good codes can be constructed by puncturing good ensembles 
of LDPC codes using a technique called "intentional puncturing". In this approach, the code bits 
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are partitioned into disjoint sets so that each set contains all the code bits whose corresponding 
variable nodes have the same degree. The code bits in each one of these sets are randomly punctured 
at a fixed puncturing rate. 

We briefly present the notation used in W for the characterization of ensembles of intentionally 
punctured LDPC (IP-LDPC) codes. Consider an ensemble of LDPC codes with left and right edge 
degree distributions A and p, respectively. For each degree j such that Xj > 0, a puncturing rate 
TTj E [0, 1] is determined for randomly puncturing the set of code bits which correspond to variable 
nodes of degree j. The polynomial associated with this puncturing pattern is 



(55) 



An ensemble of IP-LDPC codes can be therefore represented by the quadruplet (n, A,p, vr'^) where 
n designates the block length of these codes, A and p are the left and right degree distributions from 
the edge perspective, respectively, and 7r(0) is the polynomial which corresponds to the puncturing 
pattern, as given in The average fraction of punctured bits is given by p^'^^ = ^"jLi ^j'^j where 
A is the left node degree distribution of the original LDPC ensemble. The following statement, 
which relies on Theorem l4.11 provides an upper bound on the common design rate of a sequence of 
ensembles of IP-LDPC codes. This bound refers to ML decoding (and hence, to any sub-optimal 
decoding algorithm). 



Theorem 5.2. Let {{nr, X, p,ir^)}^i be a sequence of ensembles of IP-LDPC codes transmitted 
over an MBIOS channel, and assume that tends to infinity as r ^ oo. Let C be the channel 
capacity, and a be the pdf of the LLR at the output of the channel given its input is 1. If the 
asymptotic bit error probability of this sequence vanishes under ML decoding (or any sub-optimal 
decoding algorithm) as r — > oo, then in probability 1 w.r.t. the puncturing patterns, the common 
design rate of these ensembles satisfies 



1 - pio) 



l-(l-p(o))C 



2 In 2 



E 



^ lp(2p 



' ^ i=i 



(56) 



where F, as introduced in (jS)), denotes the right degree distribution from the node perspective. 



(57) 



designates the average puncturing rate of the code bits, and Qp is the functional of the MBIOS 
channel introduced in H46() . 



Proof. The proof follows from Theorem l4.H and the observation that IP-LDPC codes form a special 
case of the ensemble (n. A, p) examined in Sectional For a sequence of ensembles of IP-LDPC codes, 
{{uj., A, p, vr'^)}, the number of parallel MBIOS channels used for transmission is equal to the number 
of strictly positive coefficients in the polynomial A, i.e., J = |{i : Aj > 0}|. Denote these degrees 
hy ii, ... ,ij , then the bits transmitted over the j*^ channel are those involved in exactly ij parity- 
check equations (i.e., the bits whose corresponding variable nodes are of degree ij). From the above 
discussion, it follows that the fraction of code bits transmitted over the j^^ channel is given by 

p, = Ai^ , i G {1, . . . , J} (58) 
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and the fraction of edges in the bipartite graph which are connected to variable nodes transmitted 
of the j^^ channel is given by 

qj = Ai, , i G {1, . . . , J}. (59) 

The transmission of IP-LDPC codes over an MBIOS channel is equivalent to transmitting these code 
over a set of J parallel MBIOS channels where each of these channels forms a serial concatenation 
of a BEG whose erasure probability is equal to the puncturing rate vTj ., followed by the original 
MBIOS channel. Hence, the prf/of the LLR at the output of the j^^ MBIOS channel given its input 
is 1 gets the form 

a{l;j) = TTi,do{l) + (1 - vr,Ja(0, / G M (60) 
and the capacity of this channel is 

Cj = C{l-7ri^). (61) 
By substituting H^U^ into we get that for all j G {1, . . . , J} and p G N 

9j,P = ^"h/o(0 + (l-vri,MO] (l + e-')tanh2f (^i) 
= (l-7r,^.)^~a(0 (l + e-')tanh2f (^0 dl 

= (l-vr.,)5p (62) 

where the last equality is based on (|i6|) . The statement now follows by substituting (|58|) , (|59|) , (|61|) 
and (|62() in ()32() : we finally use the scaling factor for the design rate of punctured codes, as given 
in Definition 15 . II (in this case, the parameter 7 introduced in this definition tends to p^^^ where this 
convergence is in probability 1 w.r.t. the puncturing patterns; p^^\ as introduced in (|57|) . denotes 
the average puncturing rate of the code bits). Finally, since Xj = Aj = for j ^ {ii, . . . 
then regarding the sums in the RHS of we get the equalities ^j'^j — 'l2j=i ^ij'^ij ^^'^ 

Xlj^i Aj'TTj = X]/=i ^ij^ij- This completes the proof of the theorem. □ 
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5.4 Numerical Results for Intentionally Punctured LDPC Codes 



In this section, we present a comparison between thresholds under message-passing iterative (MPI) 
decoding and bounds on thresholds under ML decoding for ensembles of IP-LDPC codes. It is 
assumed that the transmission of the punctured LDPC codes takes place over a binary-input 
AWGN channel. The pairs of degree distributions and the corresponding puncturing patterns 
were originally presented in [IJ E] . We use these ensembles in order to study their inherent gap 
to capacity, and also study how close to optimal is iterative decoding for these ensembles (in the 
asymptotic case where the block length goes to infinity). 

We refer here to three ensembles of IP-LDPC codes: Tables ^ and |2l refer to two ensembles of 
rate— ^ LDPC codes which by puncturing, their rates vary between 0.50 and 0.91; Table |21 refers to 
an ensemble of rate— ^ LDPC codes which by puncturing, its rate varies between 0.10 and 0.83. 
Based on Lemma [2. 11 we verify that the design rates of these three ensembles of LDPC codes (before 
puncturing) are equal in probability 1 to the asymptotic rates of codes from these ensembles. This 
conclusion still holds for the punctured LDPC ensembles given in Tables (see Lemma l5.2() . 
This enables to calculate the capacity limits which refer to the design rates of these ensembles, and 
to evaluate the gaps to capacity under ML decoding and iterative decoding for these ensembles of 
punctured LDPC codes. 

For various ensembles of IP-LDPC codes. Tables ^121 provide lower bounds on the inherent gap 
to capacity under optimal ML decoding (based on Theorem 15. 2|) : these values are compared to the 
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(puncturing pattern) 


Design 
rate 


Capacity 
limit 


Lower bound 
(ML decoding) 


Iterative (IT) 
Decoding 


Fractional gap to 
capacity (ML vs. IT) 





0.500 


0.187 dB 


0.270 dB 


0.393 dB 


> 40.3% 


0.07886a: + 0.01405.T'' + 
0.06081a;3 + 0.07206a;9 


U.oZo 


U.oio aJj 


n Q07 


U.oZD Qjj 


o ( .y /o 


0.20276a; + O-OgSOSa;"^ + 
0.03356a;3 + 0.16504a3 


u.oyz 


U.Doo ar> 


U. / ID Qo 


U.oo/ Qr> 


oD.4/0 


0.25381a + 0.15000x^ + 
0.34406^3 + 0.019149^9 


u.Dzy 


U.ooD ar> 


u.yzo Qjj 


i.UDo Qjj 


\ Q7 QQ/ 


0.31767a + 0.18079a-^ + 
0.05265^3 + 0.24692a9 


V.Ui i 


1 HQQ ATi 

l.Uoo ar> 


1 1 71 ATi 

1.1(1 ar> 


i.(5oU Qjj 


OO.D/O 


0.36624a + 0.24119^ + 
0.49649a3 + 0.27318a9 


n 71 


1 '^OQ 

i.oyo QJj 




i.DD4 Qjj 


OD.y/O 


0.41838a + 0.29462x^ + 
0.05265a3 + 0.30975a9 


n 77/1 

U. M 4 


1 Q1 /I ATi 


1 007 ATi 


1 1 ATi 
Z . i i Qjj 


\ Q7 90/ 


0.47074a + 0.34447a''^ + 
0.02227a3 + 0.34997a9 


U.ooo 


9 /ino /^R 

Z.4Uy Qo 




9 781 /^R 
Z. / oi Qjj 


> Q7 1 07 


0.52325a + 0.39074x^ + 
0.01324a3 + 0.39436a'5 


0.912 


3.399 dB 


3.607 dB 


3.992 dB 


> 35.1% 



Table 1: Comparison of thresholds for ensembles of IP-LDPC codes where the original ensemble before 
puncturing has the degree distributions A(a) = 0.25105x + 0.30938a2 + 0.00104a3 + 0.43853a9 and p{x) = 
0.63676x^ + 0.36324^'' (so its design rate is equal to ^). The transmission of these codes takes place over 
a binary-input AWGN channel. The table compares values of ^ referring to the capacity limit, the bound 
given in Theorem 15.21 (which provides a lower bound on ^ under ML decoding) , and thresholds under 
iterative message-passing decoding. The fractional gap to capacity (see the rightmost column) measures the 
ratio of the gap to capacity under optimal ML decoding and the achievable gap to capacity under (sub- 
optimal) iterative message-passing decoding. The pair of degree distributions for the ensemble of LDPC 
codes, and the polynomials which correspond to its puncturing patterns are taken from |31 Table 2]. 

corresponding gaps to capacity under iterative message-passing decoding (whose calculation is based 
on the density evolution analysis). On one hand. Tables E^Hlpi'ovide a quantitative assessment of the 
loss in the asymptotic performance which is attributed to the sub-optimality of iterative decoding 
(as compared to optimal ML decoding), and on the other hand, they provide an assessment of 
the inherent loss in performance which is attributed to the structure of the ensembles, even if 
optimal ML decoding could be applied to decode these codes. The loss in performance in both 
cases is measured in terms of in decibels. It is demonstrated in Tables that for various 
good ensembles of IP-LDPC codes, the asymptotic loss in performance due to the code structure 
is still non-negligible as compared to the corresponding loss due to the sub-optimality of iterative 
decoding. As an example, for all the ensembles of IP-LDPC codes considered in Table ^ (which 
were originally introduced in 4, Table 2]), the gap to capacity under the sum-product iterative 
decoding algorithm does not exceed 0.6 dB; however, under ML decoding, the gap to capacity is 
always greater than | of the corresponding gap to capacity under this iterative decoding algorithm; 
therefore, the results in Table ^ regarding the thresholds under ML decoding further emphasize 
the efficiency of the sum-product decoding algorithm for these ensembles, especially in light of its 
moderate complexity. 

Tables ElSl also show that the performance of the punctured LDPC codes is degraded at high 
rates, where one needs to pay a considerable penalty for using punctured codes. This phenomenon 
was explained in [121 Theorem 1] by the threshold effect for ensembles of IP-LDPC codes. 

Following the performance analysis of punctured LDPC codes in [211^0^1) the numerical 
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(puncturing pattern) 


Design 
rate 


Capacity 
limit 


Lower bound 
(ML decoding) 


Iterative (IT) 
Decoding 


Fractional gap to 
capacity (ML vs. IT) 





0.500 


0.187 dB 


0.234 dB 


0.299 dB 


> 41.5% 


0.102040a; + 0.06497a;^ + 
0.06549a:5 + O.OOSSlx^ + 
0.39377a;i9 


0.555 


0.450 dB 


0.473 dB 


0.599 dB 


> 15.4% 


0.226410a; + 0.14149x^ + 
0.21268a;5 + O.OOOOla;^ + 
0.4424a;i^ 


0.625 


0.816 dB 


0.841 dB 


1.028 dB 


> 11.9% 


0.348940a; + 0.21015x^ + 
0.38902a;'^ + 0.00003a;6 + 
0.48847a;i^ 


0.714 


1.368 dB 


1.398 dB 


1.699 dB 


> 8.9% 


0.410320a; + 0.24330x^ + 
0.48388a;'^ + 0.00004a;6 + 
0.50541a;i9 


0.769 


1.777 dB 


1.811 dB 


2.215 dB 


> 7.8% 


0.469100a; + 0.28408x^ + 
0.56178a;5 + 0.00002x6 ^ 
0.53412a;i9 


0.833 


2.362 dB 


2.404 dB 


3.004 dB 


> 6.6% 


0.533750a; + 0.30992x^ + 
0.66375a;5 + O.OOOOlx^ + 
0.54837a;i9 


0.909 


3.343 dB 


3.410 dB 


4.634 dB 


> 5.2% 



Table 2: Comparison of thresholds for ensembles of IP-LDPC codes where the original LDPC ensemble 
before puncturing has the degree distributions A(a;) = 0.23403a; + 0.21242a;2 + 0.14690a;5 + 0.10284^^ + 
0.30381x^^ and p{x) = 0.71875a;'' + 0.28125a;* (so its design rate is equal to ^). The transmission of these 
codes takes place over a binary-input AWGN channel. The table compares values of referring to the 
capacity Hmit, the bound given in Theorem 15.21 f which provides a lower bound on ^ under ML decoding), 
and thresholds under iterative message-passing decoding. The fractional gap to capacity (see the rightmost 
column) measures the ratio of the gap to capacity under optimal ML decoding and the achievable gap to 
capacity under (sub-optimal) iterative message-passing decoding. The pair of degree distributions for the 
ensemble of LDPC codes, and the polynomials which correspond to the puncturing patterns are taken from 
H Table 3]. 

results shown in Tables exemplify the high potential of puncturing in designing codes which 
operate closely to the Shannon capacity limit and used for rate-compatible coding for various 
MBIOS channels. Other examples of capacity-achieving ensembles of punctured codes on graphs 
are the irregular repeat-accumulate (IRA) codes and accumulate-repeat-accumulate (ARA) codes. 
Recently, it was shown by Pfister et al. that properly designed nonsystematic IRA codes achieve 
the capacity of the BEG with bounded decoding complexity per information bit jTOj. This bounded 
complexity result is achieved by puncturing all the information bits of the IRA codes, and allowing 
in this way a sufficient number of state nodes in the Tanner graph representing the codes. This 
is in contrast to all previous constructions of capacity- achieving LDPC codes whose complexity 
becomes unbounded as their gap to capacity vanishes. 

The decoding complexity of punctured LDPC codes for parallel channels is addressed in the 
next section. 
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(puncturing pattern) 


Design 
rate 


Capacity 
limit 


Lower bound 
(ML decoding) 


Iterative (IT) 
Decoding 


Fractional gap to 
capacity (ML vs. IT) 





0.100 


-1.286 dB 


-1.248 dB 


-1.028 dB 


> 14.5% 


0.486490a; + 0.69715a;^ + 
0.03287a;3 + 0.04248x4 + 
0.69048a;^ +0.45209x^4 


0.203 


-0.953 dB 


-0.917 dB 


-0.731 dB 


> 16.3% 


0.655580a; + 0.83201x^ + 
0.48916a;3 + 0.33917a;4 + 
0.63990a;^ + 0.76947a;24 


0.304 


-0.605 dB 


-0.570 dB 


-0.317 dB 


> 12.0% 


0.745690a; + 0.87184a;^ + 
0.38179a;3 + 0.48427a;4 + 
0.74655a;^ + 0.79130a;24 


0.406 


-0.226 dB 


-0.189 dB 


+0.029 dB 


> 14.7% 


0.838470a; + 0.65105x^ + 
0.04527a;3 + 0.95233a;4 + 
0.74808a;^ + 0.80845a;24 


0.487 


+0.130 dB 


+0.171 dB 


+0.599 dB 


> 8.7% 


0.979320a; + 0.46819x^ + 
0.71050a;3 + 0.59816a;4 + 
0.79485a;^ +0.05765x^4 


0.577 


+0.556 dB 


+0.840 dB 


+ 1.152 dB 


> 47.7% 


0.895200X + 0.84401x^ + 
0.98541x3 + 0.42518x4 + 
0.92976x^ +0.30225x24 


0.663 


+ 1.039 dB 


+1.232 dB 


+ 1.806 dB 


> 25.2% 


0.910960X + 0.91573x2 + 
0.23288x3 + 0.40977x4 + 
0.99811x^ +0.15915x24 


0.747 


+ 1.605 dB 


+1.958 dB 


+2.637 dB 


> 34.2% 


0.904130X + 0.96192x2 + 
0.35996x3 ^ 0.96980x4 ^ 
0.31757x^ +0.89250x24 


0.828 


+2.303 dB 


+2.505 dB 


+3.863 dB 


> 13.0% 



Table 3: Comparison of thresholds for ensembles of IP-LDPC codes where the original ensemble before punc- 
turing has the degree distributions A(x) = 0.414936x + 0.183492x2 + 0.013002x3 + 0.093081x4 + 0.147017x^ + 
0.148472x24 and p{x) = 0.4x2 + 0.6x3 (so its design rate is equal to j^). The transmission of these codes takes 
place over a binary-input AWGN channel. The table compares values of ^ referring to the capacity limit, 
the bound given in Theorem 15. 21 f which provides a lower bound on under ML decoding), and thresholds 
under iterative message-passing decoding. The fractional gap to capacity (see the rightmost column) mea- 
sures the ratio of the gap to capacity under optimal ML decoding and the achievable gap to capacity under 
(sub-optimal) iterative message-passing decoding. The pair of degree distributions for the ensemble of LDPC 
codes, and the polynomials which correspond to the puncturing patterns are taken from 5, Table 5.1]. 

6 Lower Bounds on the Decoding Complexity of LDPC Codes for 
Parallel Channels 

The scope of this section is to derive a lower bound on the decoding complexity of LDPC codes for 
parallel MBIOS channels. The lower bound holds under message-passing iterative (MPI) decoding, 
and it grows like the logarithm of the inverse of the gap (in rate) to capacity. Interestingly, a 
logarithmic behavior of the parity-check density (which forms a measure of the decoding complexity 
per iteration) in terms of the gap to capacity also characterizes the upper bound derived in O 
Section 3]; this upper bound refers to MacKay's ensemble of LDPC codes whose transmission takes 
place over a set of parallel MBIOS channels. 

In the previous section we regarded the transmission of punctured LDPC codes over MBIOS 
channels as a particular case of the transmission of the original codes (before puncturing) over a 
set of parallel MBIOS channels. Hence, the aforementioned bound is later applied to obtain lower 
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bounds on the decoding complexity of (randomly and intentionally) punctured LDPC codes. This 
section refers to an appendix which suggests a simplified re-derivation of IIUI Theorems 3 and 4], 
and shows that the bounds introduced in this section are tighter. 

6.1 A Lower Bound on the Decoding Complexity for Parallel MBIOS Channels 

Consider a binary linear block code which is represented by a bipartite graph, and assume that 
the graph serves for the decoding with an iterative algorithm. Following [S] and ^U], the decoding 
complexity under MPI decoding is defined as the number of edges in the graph normalized per 
information bit. This quantity measures the number of messages which are delivered through the 
edges of the graph (from left to right and vice versa) during a single iteration. Equivalently, since 
there is a one-to-one correspondence between a bipartite graph and the parity-check matrix H which 
represents the code, the decoding complexity is also equal to the number of non-zero elements in 
H normalized per information bit (i.e., the density of the parity-check matrix jl51 Definition 2.2]). 
Hence, the decoding complexity (as well as the performance) of iteratively decoded binary linear 
block codes depends on the specific representation of the code by a parity-check matrix. Since the 
average right degree (or) of a bipartite graph is equal to the number of edges normalized per parity- 
check equation, then the average right degree and the decoding complexity are related quantities. 
Consider an ensemble of LDPC codes whose design rate is R^- It is natural to relate the decoding 
complexity of the ensemble, say xd, to its average right degree and design rate, as follows: 

XD = 5 OR • 

We note that or is fixed for all the codes from an ensemble of LDPC codes with a given pair of 
degree distributions. 

The following lemma will be used for the derivation of a lower bound on the decoding complexity 
per iteration under MPI decoding. 

Lemma 6.1. Let F be the right degree distribution of an ensemble of LDPC codes. Then for any 
a > it holds that 

F(a) > a"^. 

Proof. Using the convexity of the function /(x) = a^, it follows from Jensen's inequality that 

oo 

F(a) = ^Tia' > a^-i^^» = a"^. 

i=l 

□ 

Consider a sequence of ensembles of LDPC codes, {n.,.. A, p}'^i, whose transmission takes place 
over a set of J statistically independent parallel MBIOS channels. Let Cj and pj be the capacity and 
the fraction of code bits which are assigned to the j**^ channel, respectively (where j G {1, . . . , J}). 
We define the average capacity of the set of J parallel channels as C = Ylj=iPj^j- ensemble 
of LDPC codes which achieves vanishing bit error probability as the block length goes to infinity, 
the multiplicative gap (in rate) to capacity is defined as 

e^l-&. (63) 

We now present a lower bound on the decoding complexity per iteration under MPI decoding for 
this sequence. The bound is given in terms of the gap to capacity. 
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Theorem 6.1. Let a sequence of ensembles of LDPC codes, {{nr, X^,, p)}^^, be transmitted over 
a set of J statistically independent parallel MBIOS channels. Assume that the capacities Cj of 
these channels are all positive, and denote the average capacity by C = '^j=iPjCj. If this sequence 
achieves a fraction 1 — e of C with vanishing bit error probability, then the asymptotic decoding 
complexity under MPI decoding satisfies 

XD(e) >i^i+i^2lnf-^ (64) 



where _ 

(l-C)ln(^i^) 

Ki = - _ . ; K2 = - _ ^ (65) 

C In (^Ej=i (lj9j,i ) C \n \Y.j=i Qj9j,i ) 

where gj^i is introduced in (jSJ, and qj is introduced in H31|) and is assumed to be positive for all 

j G {1, . . . , J}. For parallel BECs, the term can be removed from the numerator of Ki. 

Proof. Substituting ^ in ^ yields 

{l-e)C<l ^ . ^"^ r-r T^- (66) 



^ 21n2g|p(2p-l)^(g^^-^^'^^ 



Since gj^p in Q is non- negative for j S {1, . . . , J} and p E N, and the function T is non- negative on 
M^, then the terms in the infinite sum above are all non-negative. By the truncation of this series 
where we only take its first term (note that this is the largest term in the sum), we obtain a lower 
bound on the RHS of (|66|) . This implies that 

{l-e)C<l ^"^ 



2 In 2 

Invoking Lemma l6. II vields that 



\j=l 



{l-e)C<l ^"^ 



2 In 2 

The solution of the last inequality for the average right degree (or) gives 



OR > 



^^^1 21n2 ^ Ce 



(67) 



where the last step follows by dropping the 1 which appeared inside the logarithm at the numerator 
(this step is valid since the denominator is strictly negative), and 



In 



1 i-c 



, , 2 In 2 c / / 
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Since Rd < C, it follows that yd = ^-r^ or > or- The proof of the lower bound on the 

G 

decoding complexity for parallel MBIOS channels follows by multiplying both sides of (|67j) by 
For parallel BECs, we get from © that for every p gN 



9j,p 



j\{l;j){l + e-^)tanh'^P dl = 1 



where £j denotes the erasure probability of the j*^ EEC. This gives 

oo ( ^ /J 



=— f ^rfy 

21n2^p(2p-l) 



3=1 / 



Substituting this in (|66|) . gives 

{l-e)C<l ^"^ 



Vi=i 



The continuation of the proof follows the same steps as the proof for parallel MBIOS channels, 
and leads to the improved coefficient Ki, i.e., without the factor in the numerator of Ki for 
general MBIOS channels (see (|65|>). □ 

We proceed the analysis by the derivation of lower bounds on the decoding complexity of 
sequences of ensembles of punctured LDPC codes where it is assumed that these sequences achieve 
vanishing bit error probability; similarly to Theorem 16. 11 the lower bounds are expressed in terms 
of the multiplicative gap (in rate) to capacity. 



6.2 Lower Bounds on the Decoding Complexity for Punctured LDPC Codes 

As discussed in the previous section, transmission of punctured codes can be interpreted as a 
special case of transmitting the original (un-punctured) codes over a set of parallel channels where 
these component channels are formed by a mixture of the communication channel and BECs whose 
erasure probabilities are the puncturing rates of different subsets of code bits. Hence, the bounds 
on the decoding complexity of punctured codes can be derived as special cases of the bound given 
in Theorem 16.11 For the sake of brevity, we derive these bounds by using the upper bounds on 
the achievable rates of punctured LDPC codes given in Theorem 15.11 (for random puncturing) and 
Theorem 15. 21 ffor intentional puncturing). Note that the derivation of the latter two theorems relies 
on Theorem 14.11 fas shown in Fig. ^ on p. I32|). 

Consider an ensemble of LDPC codes of length n and design rate R'^, and let the code bits be 
partitioned into J disjoint sets where the j^^ set contains a fraction pj of these bits {j G {1, . . . , J}). 
Assume that the bits in the j^^ set are randomly punctured at rate tTj , and let the punctured codes 
be transmitted over an MBIOS channel whose capacity is C . As shown in the previous section, this 
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is equivalent to transmitting the original (un-punctured) codes over a set of J parallel channels, 
where the j^^ set of code bits is transmitted over a channel whose capacity is Cj = (1 — ■Kj)C. The 
average capacity of this set of J parallel channels is therefore given by 

J J 
C = Y.P^{1- ^j)C = (1 - Y.P3^3)C = (1 - l)C 

where 7 = '}2j=iPj''^j overall puncturing rate. Denote the design rate of the punctured codes 

by i?d = (see Definition 15.11 on p. I18() . then it follows that the multiplicative gap to capacity 
of the punctured codes is given by 

e = l-f = l-f. (69) 

For punctured codes, the iterative decoder is based on the bipartite graph of the 'mother code' 
where the channel input to the variable nodes which correspond to the punctured code bits is 
defined to be 0. Hence, the decoding complexity of the punctured ensemble under MPI decoding is 
identical to the decoding complexity of the original ensemble (before puncturing), and is given by 

l-{l-7)Rd 
= il-,)R, 

In the following, we derive a lower bound on the decoding complexity of a sequence of ensembles 
of RP-LDPC codes. 

Theorem 6.2. Let {{rir, X, p)}'^i be a sequence of ensembles of LDPC codes whose block length 
(rir) tends to infinity as r — > 00. Assume that a sequence of ensembles of RP-LDPC codes is 
constructed in the following way: for each code from an ensemble of the original sequence, a subset 
of arir code bits is a-priori selected, and these bits are randomly punctured at a fixed rate (Ppct)- 
Assume that the punctured codes are transmitted over an MBIOS channel with capacity C, and 
that as r tends to infinity, the sequence of ensembles of punctured codes achieves a fraction 1 — e 
of the capacity with vanishing bit error probability. Then in probability 1 w.r.t. the random 
puncturing patterns, the decoding complexity of this sequence under MPI decoding satisfies 

XD(e) >i^i + i^2ln0) (71) 

where _ 

i-c 

C ln((l-Ppct+65i)' C ln((l-Ppct+05i) 

where gi is introduced in (|i6|) . ^ is introduced in (|l8|) . and C = (1 — aPpct)C. For the particular 
case of a BEC, the term can be dropped, thus improving the tightness of the additive term 
(Ki) in the lower bound. 

Proof. Since the code bits of a subset of the code bits whose size is an^ are randomly punctured 
at rate ipct, then the average puncturing rate is given by 7 = aPpct- Hence, Eq. yields that 
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C = {1 — aPpct)C. By multiplying both sides of (|17l) by 1 — aPpct and getting from (I^U]) that 
i?d = (1 — £)C, we obtain 



:i-e)c<i 



l-C 



oo 



21n2 ^ [p(2p - 1 



^ r((l-Ppet + 05p 



Following the same steps as in the proof of Theorem l6.H we get a lower bound on the average right 
degree of the bipartite graph which corresponds to the pair of degree distributions (A,/?). This 
lower bound is of the form 



aR>Ki + K^ln0^ (73) 



where 

In 



1 l-C 



ln((l-Ppct+05i)' ' ln((l-Ppet+05i 



Note that is positive; this follows from (|48|) . which yields that ^ < (1 — a)Ppct (due to the fact 
that the integral of A over the interval [0, 1] is upper bounded by ^). This assures that as the gap 
(in rate) to capacity vanishes, the lower bound on or scales like the logarithm of the inverse of this 
gap. 

From (inni), we get R'^ = {l — e)C < C, and therefore xd = c^R > OR- The proof of the 

lower bound on the decoding complexity is completed by multiplying both sides of ((75|) by 

In the particular case where the communication channel is a BEC, following the same concept as 

in the proof of Theorem 16. II leads to the improved coefficient Ki. □ 



The upper bound on the decoding complexity for sequences of ensembles of IP-LDPC codes is 
also given in terms of the gap between the rate of the punctured rate and the channel capacity. 

Theorem 6.3. Let {{n-r, X, p, Tr^)}^^ be a sequence of ensembles of IP-LDPC codes transmitted 
over an MBIOS channel whose capacity is C. If this sequence achieves a fraction 1 — e of the 
capacity with vanishing bit error probability, then in probability 1 w.r.t. the random puncturing 
patterns, the decoding complexity of this sequence under MPI decoding satisfies 



XD(e) >i^i + i^2ln0) (74) 
where _ 

C InM 1 - Y.%x Aj^j j gA C7 InM 1 - ^ A^-^j j g^ \ 

where g\ is introduced in (|i6|) . and C = (1 — Z^^i ^j'^'})^- For the particular case of a BEC, the 
term can be dropped, thus improving the tightness of the additive term {K\) in the lower 
bound. 



Proof. The proof follows from the same concepts as the proof of Theorem 16.21 but is based on ()56() 
instead of (|47)) . Note that K2, which reflects the logarithmic growth rate of the lower bound in 
((71]) . is always positive; this follows from ((7S)) and due to the fact that from gi < 1, and also 

o<i-EI^iA,^,<i. □ 
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6.3 Re-Derivation of Reported Lower Bounds on the Decoding Complexity 

In |in| Theorems 3 and 4], Pfister et al. introduced lower bounds on the decoding complexity of 
punctured codes on graphs with iterative decoding. The bounds were derived for the case where 
a subset of hnearly independent code bits whose size is equal to the code dimension are randomly 
punctured at a fixed rate (-Ppct), and the transmission of the codes takes place over an MBIOS 
channel. In particular, this scenario corresponds to RP-LDPC codes (see Section 15. 2() where we 
choose a subset of the code bits to be randomly punctured at rate -Ppct; under the assumption in 
jlUl Theorems 3 and 4], the fraction (a) of the code bits which are randomly punctured is equal 
to the code rate. In the appendix, we show that for randomly punctured LDPC codes, the lower 
bounds on the decoding complexity given in |10[ Theorems 3 and 4] follow from a looser version of 
the bound in Theorem 16.21 

7 Summary and Outlook 

The main result in this paper. Theorem 14.11 provides an upper bound on the asymptotic rate 
of a sequence of ensembles of LDPC codes which achieves vanishing bit error probability. We 
assume that the communication takes place over a set of parallel memoryless binary-input output- 
symmetric (MBIOS) channels. The derivation of Theorem 14.11 relies on upper and lower bounds 
on the conditional entropy of the transmitted codeword given the received sequence at the output 
of the parallel channels (see Section E}, and it is valid under optimal ML decoding (or any sub- 
optimal decoding algorithm). This theorem enables the derivation of a lower bound on the decoding 
complexity (per iteration) of ensembles of LDPC codes under message-passing iterative decoding 
when the transmission of the codes takes place over parallel MBIOS channels. The latter bound 
is given in terms of the gap between the rate of these codes for which reliable communication 
is achievable and the channel capacity. Similarly to a lower bound on the decoding complexity 
of ensembles of LDPC codes for a single MBIOS channel jl5j . the lower bound on the decoding 
complexity which is derived for parallel channels also grows like the log of the inverse of the gap to 
capacity. 

Theorem 14. II can be used for various applications which form particular cases of communication 
over parallel channels, e.g., intentionally punctured LDPC codes non-uniformly error protected 
LDPC codes and LDPC-coded modulation (see e.g., jHllSl)- In SectionlSl we use Theorem l4.1l 
for the derivation of upper bounds on the achievable rates under ML decoding of (randomly and 
intentionally) punctured LDPC codes whose transmission takes place over an MBIOS channel. It 
is exemplified numerically that for various good ensembles of IP-LDPC codes, the asymptotic loss 
in performance due to the code structure is still non-negligible as compared to the corresponding 
loss due to the sub-optimality of iterative decoding (as compared to optimal ML decoding). Looser 
versions of the bounds derived in this paper for punctured LDPC codes suggest a simplified re- 
derivation of previously reported bounds on the decoding complexity of randomly punctured LDPC 
codes (see Theorems 3 and 4]). 

Interconnections between the theorems introduced in this paper and some other previously 
reported results are shown in Fig. ^ (see p. IH^. 
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Information-theoretic bounds: 
Proposition 3.1 and Lemma 3.2 




Aclnievabie rates of LDPC codes 
for paraiiei I^BIOS cfianneis 

Tfieorem 4.1 


»• 




Upper bounds on tlie aclnievabie 
rates of punctured LDPC codes 
for MBIOS cfianneis 

Tfieorems 5.1 and 5.2 



Upper bound on tfie 
acfiievable rates of LDPC 
codes for IVIBiOS cfianneis 

[17, Corollary 4.1] 



Upper bound on tfie 
acfiievable rates of LDPC 
codes for parallel BECs 

[11, Theorem 2] 



Upper bound on the rates 
of LDPC codes based on 
two-level quantization 

[1 , Theorems 1 and 2] 



BEC 



Upper bound on the rates 
of LDPC codes based on 
two-level quantization 

[15, Theorem 2.5] 



Lower bounds on the 
parity-checl< density 

[15, Theorem 1] 



Lower bound on the 
decoding complexity of LDPC 
codes for paraiiei channels 

Theorem 6.1 



Lower bound on the 
decoding complexity of 
randomly punctured LDPC 
codes (Theorem 6.2) 



Upper bound on the 
achievable rates for a 

uniform BEC: 
[13, Theorem 3] and 

[15, Eq. (36)] 



Lower bound on the 
decoding complexity of 
Intentionally punctured 
LDPC codes (Theorem 6.3) 



Lower bounds on the 
decoding complexity of 
randomly punctured LDPC 
codes for MBIOS channels 
[10, Theorems 3 and 4] 



Figure 1: An interconnections diagram among the bounds in this paper and previously reported 
bounds. 
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Appendix: Re-derivation of [10. Theorems 3 and 4] 

In the following, we start with the re-derivation of |in[ Theorem 4] for general IVIBIOS channels, 
and then re-derive the refined bound in |101 Theorem 3] for a BEC. For the re-derivation of |1U1 
Theorems 3 and 4] we rely on Theorem 16.21 whose derivation is based on Theorem 15.11 Hence, 
we first loosen the upper bound on the achievable rates given in ()47l) . and then re-derive |1U1 
Theorem 4] as a consequence of this looser version. The loosening of (|47|) is done by replacing 
the positive parameter ^ introduced in (|48|) to zero, and then using the lower bound on F from 



32 



Lemma l6. II This gives 



Rd < 



1 - aR 



pet 



1 - (1 - aPpct)C 



V 



(A.l) 



Finally, truncating the infinite series in the RHS of (|A.1|1 by only taking its first term which 
corresponds to p = 1 further loosens the upper bound on the achievable rates, and gives 



1 



1 - aR 



pet 



1 



V 



1 



From we get the inequality 

(l-e)(l-aPpct)C< 1 

which after straightforward algebra gives 



1 - (1 - aPpct)C 

^((l-^'pet)^/l. 



1 - (1 - aPpct)C 

\ OR ' 



(A.2) 



2 In 2 



1-R 



pet)^! j 



1 - (1 - aPpetlC 

1 _| 1 P£iZ < 2 In 2 

eC(l-aPpet) - V(l-^pct)5i 



(A.3) 



We proceed by giving a simple lower bound on gi. 
Lemma A.l. For gi introduced in (|46() . the following inequality holds 

51 > (1 - 2w)^ 

where 

1 

w = Pe{a) = - Pr(L = 0) + / a{l) dl 
designates the uncoded bit error probability of the MBIOS channel given the channel input is 1. 

Proof. Based on the symmetry property where a{l) = e^a{—l) and Jensen's inequality, we get 

gi = a(0 (l + e-')tanh2 



> 



j a{l) tanh^ dl 
J a(OtanhQ^d/^ 
^ a(/) (1 + e-')tanh d/^ 
o(/) (1 - e-^)dl 



0+ 



{a{l) - a{-l))dl 



^1 - Pr(L = 0) - 2 J a{l)dl^ 



(1 - 2wf 
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□ 



Dropping the 1 in the LHS of (|A.3|) and replacing gi in the RHS of (|A.3|) by its lower bound 
from Lemma I A. II gives 

1 - (1 - aPpct)C / 1 ^ 

^ ^ ' < 21n2 -- 

2aR 



eC(l - aPpet) - V (1 - ^pct)(l - 2w) 

1 



< 2 In 2 



(1 - Ppct)(l - 2w) 



Solving the last inequality for or gives 

OR > 



, / 1 l-(l-aPp,t)C 
'■^^ I 2 In 2 £C(l-aPpct) 



2 In 



(l-Ppct)(l-2«;), 



Based on the equality (|7()j) which relates the complexity under MPI decoding to the average right 
degree (or) and since i?d < C, we get from the last inequality 

In f 1 l-(l-aPpct)C \ 
, , ^ 1-C ^'M21n2 eC(l-aPpct) J , , 

XD(e) > ^ W , / • 

^'^i,(i-ppct)(i-2,«); 

Note that a = Rd in |lfll Theorem 4]. This gives the equality a = (1 — — aPpct)C whose 
solution is 

Finally, the substitution of a in (|A.5|) into the RHS of ()A.4|) gives 



XD(e) > 



1„ (1 l-(l-Ppct)C+£CPpc 

1-C ^"li 2Ckr2 



2C 



In 



(l-Ppct)(l-2u>) 

1 l-(l-Ppct)C 



1-C I'^U 2C71n2 



) 



^'^l,(l-Ppct)(l-2«;); 

which coincides with [lU' Theorem 4] for a sequence of ensembles of randomly punctured LDPC 
codes. 

For the derivation of the refined bound for the BEC which is given in |lfll Theorem 3] , we start 
from (jA.ll) . The refinement of the latter bound is due to the fact that for the BEC, gp in H46|) is 
independent of p, and is equal to = 1 — Prec where Prec designates the erasure probability of 
the BEC. From (fXTf) . we get the following upper bound on the achievable rates: 



1 - oPp„ \^ 1 - ((1 - Pp=.)(l - Pbec) 



an. 



which follows from the equality Yl"^! 2p(2p-i) ~ Substituting = {I — e)(l — Prec) and the 
a in (|A.5|) gives a lower bound on or. Finally, the lower bound in |1U1 Theorem 3] follows from the 
resulting lower bound on or and the inequality xd{£) > or- 
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